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The generalization of Sliannon's theory to include messages with given autocorrelations is presented. The 
analytical calculation of the channel capacity is based on the transfer matrix method of the effective ID Hamil- 
tonian. This bridge between statistical physics and information theory leads to efficient Low-Density Parity- 
Check Codes over Galois fields that nearly saturate the channel capacity. The novel idea of the decoder is the 
dynamical updating of the prior block probabilities which are derived from the transfer matrix solution and 
from the posterior probabilities of the neighboring blocks. Application and possible extensions are discussed, 
specifically the possibility of achieving the channel capacity without compression of the data. 



Digital communication, which is the driving 
force of the modern information revolution, deals 
with the task of achieving reliable communication 
over a noisy channel. A typical communication 
channel is depicted in figure [2 (where K < L and 
K<N). 

Shannon, in his seminal workjlj, proved that in 
order to overcome noise, redundancy must be added 
to the transmitted message. The channel capacity, 
which is the maximal rate (R = ^) is a function of 
the channel bit error rate (/), bit error rate (Pb) and 
the a priori bit probability (P) 



R = 



l-H2(f) 

H2(P)-H2(Ph) 



(1) 



where H2(x) = -ji:log2(x) - (1 -x)log2(l - x). 
The entropy, defined as the information content (in 
bits per symbol) of the message, is unity for unbi- 
ased (P = 0.5) messages and decreases with bias 

(//2(0)-i/2(l) = 0). 

To maximize channel throughput, the message is 
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Figure 1 : Digital Communication Charmel 



first compressed using algorithms (e.g., |2]) which 
approach entropy for large blocks, and then encoded 
assuming a unbiased message. Typical messages 
exhibit low autocorrelation coefficients (decreasing 
with the size of the message)!^]- For instance, the 
discrete periodic 2-point autocorrelation coefficient 
of a sequence {Xj} is defined by 



1 



(i+k) mod L 



(2) 



Compressible data exhibits enhanced autocorre- 
lation coefficients, whose distribution has been ex- 
tensively studied in numerous fields (e.g., linguis- 
tics, DNA research, heartbeat intervals, etc.)j4l|3l- 

In this paper we raise two questions: 

• what is the channel capacity of correlated se- 
quences? 

• are these bounds achievable by an encoder 
alone, without the preceding compression 
phase? 

We address these questions by using models from 
statistical mechanics, which lead to a new class 
of decoding algorithms. The decoder nearly ap- 
proaches the capacity expected for perfect compres- 
sion, without directly applying any compression. 

The motivation for direct transmission of the un- 
compressed data is to overcome many difficulties, 
including: 
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• a single bit error in a compressed block would 
render the entire block useless. This is partic- 
ularly important for applications where min- 
imal distortion can be tolerated (e.g., au- 
dio/video transmission) 

• compression of different packets of the same 
length would generate different length output 
messages, calling for greater complexity of 
the encoder 

• the compression/decompression add delay to 
each transmission, and increase the complex- 
ity of the transmitter/receiver. 

We start by calculating the entropy of correlated 
bit sequences. This is done by binning the source 
into Ko bits, where Kg is the highest autocorrela- 
tion coefficient taken, and using the transfer-matrix 
method. We proceed by integrating the results of 
the transfer matrix into a new decoding algorithm, 
based on a Low-Density Parity-Check Codes (LD- 
PCC) over finite fields {GF {q))\6]. 

For the sake of simplicity, we begin by demon- 
strating the entropy calculation for two autocorrela- 
tion coefficients, namely, C\ and Co. The entropy 
of a set of binary sequences of length L is defined 
by the log of the number of possible sequences (Q) 
divided by L. Calculating O with the correlation 
constraints (Ci, C2)|7], is done by taking the trace 
over all possible states of the sequence obeying the 
constraints imposed as delta functions 



■jXj+l 



-CiL 



-C2L 



V J 



(3) 

where periodic boundary conditions are assumed 
for the indices. Using the Fourier representation of 
the delta functions and rearranging terms, one gets 



Q 



— rr 



dyidy2e 



-iL(yiCi+y2C2) 



where B is given by 



(27Tr 

L 

2 



(4) 
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Figure 2: Phase space for sequences with Ci,C2. Out- 
side the triangle (|Ci| > the entropy is zero. The 
parabolic curve is C2 = Cj. 



+y2{xi)X2 + xix^) (5) 

Using the transfer-matrix method|8], we group the 
sequence into blocks of 2 bits, obtaining the 4x4 
matrix 



y(yi,y2) 



I „2vi+2y2 



e 



2yi+2y2 
-2v, 



e '- 

^-2vi+2v2 



e 

„2yi+2y2 



(6) 

representing all possible interactions between 
neighboring blocks. We proceed by replacing the 

L 

trace with /ij , where A,„ax is the principal eigen- 
value of y (3'i,3'2)- Using the method of Laplace 
integrals we obtain for the leading order^J of Q 



-L(y* Ci +y*_ C2-\_\n ^„„C< .>'2* )) 



(7) 



where (y*,}'*) solution of the saddle point 

equations. The entropy in bits is given by 



H(CuC2) = 



j\nA„,,[y*,y*)-y*Ci - y*C2 



In 2 



(8) 

The entropy of the correlated sequences are 
shown in figure |2l for all possible {C],C2} tuples. 
The area of non-zero entropy is depicted in fig- 
ure IJjbetween the two straight lines, i.e., K"^' < 



Ci < ^-Y^- The paraboUc line is C2 - C\, where 
the entropy reduces to the case of a single Ci con- 
straint, with the entropy S - H2 (^^)Li^- This is 
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the typical case for the Ci constraint, since C2 can 
be viewed as multiplying two consecutive Ci pairs, 
hence it has the highest entropy for a given Ci . 

Note that at the boundary of the phase space the 
entropy falls abruptly to zero (a first order phase 
transition). Another point, which might appear 
counter-intuitive, is the fact that the entropy of the 
constraint {Ci - A, €-> = Z?} is not the same as 

=B,C2=A} mil- 

The transfer-matrix solution shows that the en- 
semble of sequences obeying the correlation con- 
straints are obtained from the thermal equilibrium 
solution of the 1 -D Ising spin model (where the two 
states of the spin correspond to the binary values 
of the bit {0,1} ^ {1,-1} |7]). The interaction 
length is the same as the correlation length, and the 

interaction strength is 7, — (where yS is the in- 
verse temperature). The corresponding Hamiltonian 

is H - J] XiXi+\ -'-fYi XiXi+2- This physical ob- 
servation is the key leading to our novel decoding 
algorithm. 

Testing the physical mapping, we choose a pair of 
constraints (Ci , C2), solve the transfer matrix model 
to obtain y*,)!*, then select any temperature (e.g., 
yS = 1) which gives the interactions Jt . 77 LIZI . and 
perform Monte-Carlo simulations, and indeed the 
system settles into an equilibrium state obeying the 
initial autocorrelation constraints. 

The division into blocks of 2 bits, amplifies the 
fact that interactions affect only neighboring blocks, 
and the probability of finding a block in one of its 
four possible states depends only on the state of its 
two neighbors. Labeling the left, center and right 
blocks by /, c, r respectively, one gets for the joint 
probability of three consecutive blocks 



P (c, I, r) = 



where 



S I (c) q'^S L (I, c) q'j^S R (r, c) 
Tr|c,/,/-| Si (c) q[SL(l, c) q'^S r (r, c) 



(9) 



S,(c) = ey^'"' 

SR(r,c) = gVf'^jn+yJCe.n+e^r^) 

The first term in Equation [TO] denotes the inner- 
block interactions, and the other terms denote the 



inter-block interactions. q'jqR is the posterior prob- 
ability of finding the left/right block in a specific 
state given by the decoding algorithm below, and the 
subscripts 1,2 denote the bit number in the block. 

The physical spin model emphasizes the dramatic 
difference between the case of biased bit probabil- 
ities, equivalent to an induced homogeneous field, 
where each spin is updated independently, to the 
case of correlations induced by inter-spin interac- 
tions. 

The entropy of the class of correlated sequences, 
derived by the transfer-matrix method, can be 
plugged into equation [2 Using shannon's lower 
bound, gives the channel capacity of sequences with 
two autocorrelation coefficients 



l-H2{f) 



H2(CuC2)-H2iPb) 



(11) 



The previous formulation is easily extended to 
higher autocorrelation coefficients. The procedure 
involves adding more delta functions to equation|3j 
resulting in a larger transfer matrix. For C/ , be- 
ing the highest autocorrelation coefficient, one gets 
/ variables {yi . . .yi), and a transfer matrix's dimen- 
sion is 2'x2', which is solved numerically. The sim- 
ulation results, shown in this paper, were conducted 
on sequences with 2 (Ci , C2), and 3 (Ci , C2, C3) au- 
tocorrelation constraints. 

Equation|9]is the crux of our decoding algorithm, 
it enables the decoder to set prior probabilities for 
each block based on the current state of its two 
neighbors. 

Decoding of the autocorrelated sequences is 
based on LDPCC which have been shown to asymp- 
totically nearly saturate Shannon's bound ll3l Il4l 
These codes easily lend themselves to decod- 
ing blocks of t bits by moving from boolean algebra 
to Galois-Field (GF (qYwith q = 2'). This method 
was originally studied fo], as a means of increasing 
the degree of the nodes (connectivity of the graph) 
without introducing loops, which are known to de- 
grade the decoder's performance. 

The decoding consists of iterating two rounds 
of message passing (also known as belief- 
propagation), which update two probability matri- 
ces Rm„,Qm„ (a e 0..(q- 1) stands for the state 
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of the (to, n) element, where to 6 fl ■ • ■ 7}^ ™d 
« e {1 . . . ^} is the block number lUl ). 

• Updating /?„,„ is based on the values of Qmn, 
and the specific parity check. This stage is 
left unchanged. 

• Updating Q„,„ is based on R„„ and the a priori 
probabilities of the q states of the block 

Q"^„ ^ a,„„P"„ Yl R"jn (12) 



where Umn - 
stant. 



JeM(n)\m 



is a normalizing con- 



In this stage the a priori probabilities, P^j, of the 
block, are obtained by marginalizing the joint prob- 
ability (equation|9} 
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Figure 3: Pb (decoder bit error rate) vs. / (channel bit 
error rate) with rate R = \ for uncorrelated sequences; 
for sequences with Ct = C2 = 0.72; and for sequences 
with Ci = -0.65, C2 = 0.6, C3 = -0.55 . Results of 
simulations using KS codesi 1?], with N = 10* (9,999 for 
GF(8)), averaged over at least 2,000 samples, are given 
for the regular algorithm (♦) and for the new algorithm (■ 
and • for the 2 and 3 autocorrelations respectively). 
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Siic) 



1 -I 

q''g^SR{r,c) 



(13) 



where l/r denotes the state of the « - 1/n + 1 block 
respectively, and <7^ /^^ are their posterior probabil- 
ities. The meaning of equation [T3l is summing the 
weighted contributions from all possible states of 
the two neighboring blocks to a possible state of the 
block in question. Thus, in each iteration the prior 
probability of a block is dynamically updated fol- 
lowing its neighboring blocks. 

The decoder performs the iterations until the 
message is decoded (all checks are satisfied), or one 
of two alternate halting criteria is met: (a) the max- 
imal number of iterations is reached. In this paper 
the maximum was set to 500; (b) the decoder has 
not modified its estimate of the message in the last 
50 iterations. 

Simulations were run using the Binary Symmet- 
ric Channel (BSC), and the construction of the 
check matrix (H) used is based on KS Codes iT?!, 
for /? = |, where the non-zero elements of H are 
randomly drawn from the range 1 . . .(q - 1). 

Figure |3l shows the results of simulations for un- 
correlated messages, and messages with 2 and 3 



autocorrelation constraints! 17 1. The performance 
of all 3 cases is similar, both in the number of it- 
erations, and in the distance from their respective 
limits 111 [11. 

Although we demonstrated the ability to increase 
the possible noise rate the decoder can handle while 
keeping the rate fixed, the converse is also possible. 
Keeping the channel noise fixed, higher rates are 
achievable. Turning to figure [2 the rate is defined 
as R = ^, but our encoder compresses the messages 
as well, thus the achievable rate is actually greater 
For Ph - 0, the total rate is R* = ^ = (where 
H {X) is the entropy of the message); for H {X) < R 
rates greater than 1 are feasible. 

The complexity of the encoding process remains 
linear {O (N)), since the calculation of a small num- 
ber of autocorrelation coefficients is linear. The de- 
coder's complexity per iteration scales as jq^u for 
a LDPCC decoder over GF (q), with ^ blocks, and 
M checks per block|6]. Our decoder adds an order 
of J (^q^ + q^ operations for the calculation of the 
block prior probabilities, calculating q probabilities, 
based on the pairs of q states of the 2 neighboring 
blocks, in equation^] The algorithm lends itself to 
parallel implementation which can reduce the com- 
plexity to 0(1). 
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All simulations shown in this paper were done 
assuming BSC, however the formulation is chan- 
nel independent, and can be applied to any channel 
(e.g., the popular Gaussian channel). The method 
can be also applied to different codes as well (e.g., 
Turbo-Code), by taking into account the dynami- 
cally updated block probabilities. 

The method described in this paper can eas- 
ily be extended to higher order correlation func- 
tions (e.g., 3-point correlation function, i.e., Ckh - 
Z ^\=\ ^i^i+kXi+k,, with periodic boundary condi- 
tions). Adding more correlations has the effect of 
reducing the entropy by lifting the degeneracy asso- 
ciated with 2-point correlation functions. The size 
of the transfer-matrix, however, does not have to in- 
crease, thus the decoding complexity is unchanged. 
For example, a transfer matrix of two bits can ac- 
commodate 1 additional autocorrelation - C12, and 
a transfer matrix of 3 bits can accommodate 3 addi- 
tional autocorrelations - C12, C13, C123, the last one 
being a 4-point correlation function|20|. 

We have demonstrated the ability to nearly sat- 
urate Shannon's limit without compression mech- 
anisms. The algorithm can be used for utilizing 
specific noise statistics as well. This work paves 
the way for additional extensive theoretical research 
and various practical implementations. 
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